Home

Row

Items

4

Factors

1

Subjects

350

Row

Presentation

Welcome to the data analysis of the General Cognitive Flexibility Scale (GCF or nflex). Here, you’ll find everything that was done to assess if the scale is suitable for futur use. Using R Markdown and the flexdashboard package, we provide all the graphs, table and analysis that was executed, along with a dataset of simulated data to reproduce the same analysis.
All participant gave their agreement for us to use those data, as long as they couldn’t be identified by any means.

The dataset contained in this folder is setup as follow :
- A row = an observation
- A column = a variable

Row

What’s inside ?

a. All analysis performed with relevent information and statistics
b. Steps for scale construction and validation
c. Reference
d. Dataset:
1. Age,
2. Sexe,
3. Professionnal situation,
4. Response to the Autism spectrum quotien scale’s subset named attention switching also named att_switching_X in the dataframe (Baron-Cohen et al., 2011)
5. Response to the cognitive flexibility scale named cfs_X in the dataframe (Martin & Rubbin, 1995)
6. Response to the Resistance to Change Scale’s Subset named Cognitive Rigidity also named cognitive_rigidity_X in the dataset (Oreg, 2003)
7. Response to the General Cognitive Flexibility Scale also named nflex_X in the dataframe (work-in-progress ; Weiss & Chene, 2020)

Row

Références:

  1. Martin, M. M., & Rubin, R. B. (1995). A New Measure of Cognitive Flexibility. Psychological Reports, 76(2), 623‑626. https://doi.org/10.2466/pr0.1995.76.2.623

  2. Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., & Clubley, E. (2001). The Autism-Spectrum Quotient (AQ) : Evidence from Asperger Syndrome/High-Functioning Autism, Malesand Females, Scientists and Mathematicians. Journal of Autism and Developmental Disorders, 31(1), 5‑17.

  3. Oreg, S. (2003). Resistance to change : Developing an individual differences measure. Journal of Applied Psychology, 88(4), 680‑693. https://doi.org/10.1037/0021-9010.88.4.680

  4. R Core Team (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.

Steps

Step 1 : Build your items

[1] "To generate item for a scale, you can either build them yourself or get inspired by other scales. In either way, you first need to read a lot of scientific article on your subject. You then need to define your topic, what you want to measure, with what kind of Scale (likert, thurstone, ...) and how many item you want at the end of all analysis. It is recommanded to generate 10x more item than what you want, as many will be discarded during the validation process."

Step 2 : Evaluated the item with a pre-test (N = 74)

The goal here was to check if the use of a 7 Likert item scale was good for our items. We also checked if some item were really bad. During this pre-test, subjects told us that Q5 was really to hard to understand. We therefore removed it for futur analysis. We can also see that a 7 Likert item Scale isn’t suitable. There isn’t a lot of response for the middle choice (as we can see on thos histograms with the blue line). We couldnt delete other items by looking at the distribution, because the Likert Scale Wasn’t good enough. We choose to dial back and take a 6-point Likert Scale, to force a polarity answer and avoid the middle choice pit problem.






















Step 3 : Build a new questionnaire (N = 350) and stating hypothesis

During this step, we changed the 7-point Likert Scale to a 6-point Likert scale. We also removed the Q5 item of the pretest, as it was labelled as “not understandable”. Therefore, the Q5 item in the new questionnaire doesn’t refer to the same sentence. We also put 3 other scales (resistance to change subscale, autisme quotient subscale, cognitive flexibility scale), and added at the end an url to a numerised WCST task.

Hypothesis :
Our nflex (alias GCF) scale should be highly correlated with the WCST. Resistance to change scale should be negatively correlated, autism quotient subscale should be positively correlated and cognitive flexibility scale shoudl not be correlated. We made thos assumptions based on scientific litterature (neuropsychological report, social psychologic reports, cognitive psychologic report) using pubMed and PsychInfo database. We also expect to have a 1-factor structur but need to use EFA instead of CFA, as nobody validated the nflex scale before.
























Step 4 : Assess Items distribution and discrimination

This step is used to assess the distribution of each item. We added a skewness and kurtosis analysis to check if those distribution had a “normal-like” format. Each item with more than 20% of extreme values answers must be deleted. It translates to a bad item with ceiling effect or bad discrimination power.





























Step 5 : Assess Items construct using an EFA

This is the EFA construct that shows items and their loading in the factor found with parallel analysis. Here, the reliability calculated with Omega was moderate-low (.57), as well as for the Cronbach’s Alpha (.57).



























Step 6 : Check for other scales validated construct using CFA

scale TLI RMSEA SRMR Chi2 p.value GFI AGFI CFI
CFS 0.8671831 0.1159691 0.0931220 307.4565 0.0000000 0.9680991 0.9261553 0.8913316
AQ 0.5884802 0.1332097 0.1177515 251.7531 0.0000000 0.9414338 0.8745010 0.6799291
RTC 0.9337185 0.1505368 0.0541501 17.8176 0.0001352 0.9948545 0.9331087 0.9779062

The same criterion as the EFA are used. The TLI index must be higher than .95, the RMSEA must be lower than .10, the SRMR must be lower than .08, the chi2 should be non significant, the GFI index should be higher than .95, the AGFI index must be higher than .90 and the CFI index must be higher than .90

REMINDER : All the items of each scales are fitted on 1 factor, as recommanded by the authors.


























Step 7 : Assess the validity of the GCF scale by checking correlations with other measures

The correlation analysis is used to assess the scale validity. We hypothesis that our scale construct would be positivility correlated with the AQ subscale (attention switching), negatively correlated with the WCST and cognitive rigidity subscale, and that there would be no correlation between our scale and the CFS.


























Data

Row

Comment

This page shows the raw datatable named data that contains the data used for the analysis, the descriptive statistics of this table, and some graphics that would help for a more indepth comprehension of the sample and its specificity.

Row

Descriptive statistics

Raw data table

Row

Sexe Histogram

Situation Histogram

Row

Sexe Counting

value frequency
Autre 2
Femme 274
Homme 111

Situation Counting

value frequency
Etudiant 161
Sans emploi 37
Travailleur 189

Item description

Row












ITEM DESCRIPTIVE ANALYSIS

Description

This page shows histograms of the 4 scales we used in our research. We analysed the General Cognitive Flexibility Scale to check if any items needed to be removed. Conclusion of this analysis is stated below :

The Table 1 shows items with more than 20% of extreme values. Items 1, 4, 6, 10 seems to have ceiling or skewed distribution. It is then recommended to check if extreme values of each item (values 1 and 6) have more than 20% of the responses. We therefore check for skewness and kurtosis. (Garin, 2014 ; Benzina, 2019)

Reminder :
The skewness coefficient is a third-order measure of the central tendency of the sample. A negative coefficient translate to a biased distribution toward the right. A positive coefficient translate to a biased distribution toward the left. \[Skewness \space equation : \gamma_{X} = E\left[\left(\frac{X - \mu}{\sigma}\right)^3\right]\]
The kurtosis coefficient is a 4th order measure of the central tendency of the sample. It translate to how fast the distribution goes back to 0 (if it does). Therefore, it shows if there is more extreme values than it should. A high kurtosis translate to a higher number of extreme values. \[Kurtosis \space equation : K[X] = E\left[\left(\frac{X - \mu}{\sigma}\right)^4\right]\]



Row

General Cognitive Flexibility Histogram

Table 1

item skewness kurtosis
nflex_1 -0.57 3.02
nflex_2 -0.01 2.34
nflex_3 0.09 2.41
nflex_4 -0.38 2.80
nflex_5 -0.22 2.47
nflex_6 -0.23 2.25
nflex_7 0.05 2.47
nflex_8 -0.34 2.81
nflex_9 0.15 2.37
nflex_10 -0.40 2.55

References

Benzina, N (2019). Évaluation de la flexibilité cognitive dans le trouble obsessionnel compulsif : étude de la validité de deux auto-questionnaires comparés à une tâche expérimentale. Médecine humaine et pathologie.

Row

Attention Switching subscale

Table 2

item skewness kurtosis
att_switch_1 0.52 3.11
att_switch_2 0.00 2.40
att_switch_3 -0.54 2.73
att_switch_4 0.11 2.12
att_switch_5 -0.35 2.30
att_switch_6 -0.56 3.02
att_switch_7 -0.51 2.73
att_switch_8 -0.46 2.69
att_switch_9 0.40 2.38
att_switch_10 0.12 2.15

Reference

Sonié, S., Kassai, B., Pirat, E., Bain, P., Robinson, J., Gomot, M., Barthélémy, C., Charvet, D., Rochet, T., Tatou, M., Assouline, B., Cabrol, S., Chabane, N., Arnaud, V., Faure, P., & Manificat, S. (2013). The French Version of the Autism-Spectrum Quotient in Adolescents : A Cross-Cultural Validation Study. Journal of Autism and Developmental Disorders, 43(5), 1178‑1183.

Row

Cognitive Rigidity Subscale

Table 3

item skewness kurtosis
cognitive_rigidity_1 -0.36 2.81
cognitive_rigidity_2 0.02 2.59
cognitive_rigidity_3 0.00 2.74
cognitive_rigidity_4 -0.33 3.18

Reference

Oreg, S. (2003). Resistance to change : Developing an individual differences measure. Journal of Applied Psychology, 88(4), 680‑693. https://doi.org/10.1037/0021-9010.88.4.680

Row

Cognitive Flexibility Scale

Table 4

item skewness kurtosis
cfs_1 -0.40 2.63
cfs_2 -0.31 2.61
cfs_3 -0.54 2.58
cfs_4 -0.21 3.22
cfs_5 -0.43 2.81
cfs_6 -0.32 2.51
cfs_7 -0.14 3.23
cfs_8 -0.24 3.02
cfs_9 -0.08 3.12
cfs_10 -0.53 2.99
cfs_11 -0.69 3.13
cfs_12 -0.23 2.80

Reference

Martin, M. M., & Rubin, R. B. (1995). A New Measure of Cognitive Flexibility. Psychological Reports, 76(2), 623‑626. https://doi.org/10.2466/pr0.1995.76.2.62 3

EFA

Row

Visualisation of Parallel Analysis

Visualisation of EFA Construct

Visualisation of Item Loadings

Row

Description

We used the parallel analysis with polychoric correlation because we have ordinal data, skewed items and non normality.We then used the efa with least squares algorithm (principal axis factor analysis), with number of factor indicated by the parallel analysis, and a varimax rotation. We set the factor loadings limit to .40 (Peterson, 2000). Anything below is considered too small.

Fit index of the EFA shows how well the model is doing compare to a base model (all item in one factor). It also shows the fitting of items on the factors. We need to check for the \(\chi²\) test (results close to 0 shows a perfect fit), the root meant square residuals (range between 0 and 1), the standardized root mean residual (SRMR), the Tucker Lewis index (TLI).The \(\chi²\) test is used for hypothesis testing to evaluate the appropriateness of a structural equation model. It checks if the sample covariance matrix \(S\) is equal to the model-implied covariance matrix \(\Sigma (\theta)\) (Null hypothesis :\(S-\Sigma (\hat{\theta})=0\)). The \(\chi²\) test is sensitive to number of observation. He will always be significant with more than 400 observations. (Schermelleh-Engel et al., 2003).

Because exact fit never occurs, the null hypothesis of exact fit is replaced by the null hypothesis of “close-fit”. Thus, the RMSEA is a measure of approximate fit in the population and is therefore concerned with the discrepancy due to approximation. (Schermelleh-Engel et al., 2003). Steiger (1990) and Browne and Cudeck (1993) define a “close-fit” as a \(RMSEA \leq .05\), an adequate fit as a \(.05\leq RMSEA \leq.08\), a mediocre fit as \(.08\leq RMSEA \leq.10\) and anythin else as not acceptable. For Hu and Bentler (1999), a cutoff of .06 is appropriate. RMSEA is relatively iundependent of sample size and favors parsimonious models (Browne and Cudeck, 1993 ; Kaplan, 2000).
The standardized root mean square residual (SRMR) was developped to overcome the problems that comes along with the root mean residual, which is dependant on the siezs of the variance and covariances of the observed variables.A value of 0 indicates a perfect fit. But there is not real cutoff, as this value is still dependent of variance and covariances of the observed variables (even if less than for the RMR). Hu and Bentler (1995) suggested that \(SRMR \leq .05\) indicate a good fit and \(.05\leq SRMR \leq.10\) indicates an acceptable fit.
The Turker Lewis Index, also known as NonNormed Fit index (Bentler and Bonnett, 1980) ranges from 0 to 1 but can sometime go beyond 1, as it is nonnormed. Less restrictive models (more complexe) are penalized while more persimonious models are rewarded by an increase in the fit index. This index is one of the less affected by sample size (Bentler, 1990 ; Hu et Bentler, 1998 ; Schermelleh-Engel et al., 2003)
The Omega is an estimation of the general factor saturation in a scale. The Omega asymptotic coefficient can be compare to a Guttman \(\lambda^6\) (or the Cronbach \(\alpha\)).

Polychoric Correlation

nflex_2 nflex_3 nflex_5 nflex_7 nflex_8 nflex_9
nflex_2 1.0000000
nflex_3 -0.2200531 1.0000000
nflex_5 -0.1643719 0.2425024 1.0000000
nflex_7 -0.2792334 0.3078536 0.2796122 1.0000000
nflex_8 0.2160019 -0.0335953 0.0472469 -0.1530191 1.0000000
nflex_9 -0.0555067 0.1449438 0.0841761 0.4194474 -0.1042351 1

Assumptions

  1. With the 6 items (2,3,5,7,8,9)
    \(MSA_{nflex} = 0.63 \space (MSA_{min} = 0.57)\) .. KMO
    \(\chi^2(15) =205.59,\space p <0\) .. Bartlett Sphericity Test.
    \(det(A) = 0.55 ,\space non-identity\space matrix\)
    2 items have a bad MSA and must be deleted. All the assumptions must be calculated again

  2. With the 4 items (2,3,5,7)
    \(MSA_{nflex} = 0.68 \space (MSA_{min} = 0.65)\) .. KMO
    \(\chi^2(6) =110.52,\space p <0\) .. Bartlett Sphericity Test.
    \(det(A) = 0.73 ,\space non-identity\space matrix\)
    All items are good for an EFA Analysis

Cut-off Criterion

TLI RMSEA SRMR BIC Chi2 p.value
RMSEA 1.040851 0.0000000 0.0187957 -11.27563 0.4945884 0.7809109
lower 0.0000000
upper 0.0662668
confidence 0.9000000

Loadings

Loadings Communalities R²
nflex_2 -0.45 0.21 0.10
nflex_3 0.53 0.28 0.13
nflex_5 0.44 0.20 0.10
nflex_7 0.63 0.40 0.16

Residuals observed correlation matrix

nflex_2 nflex_3 nflex_5 nflex_7
nflex_2 0.79 0.00 0.02 -0.01
nflex_3 0.00 0.72 0.01 -0.01
nflex_5 0.02 0.01 0.80 0.00
nflex_7 -0.01 -0.01 0.00 0.60

Reliability

alpha Omega
nflex reliability 0.57 0.57

Remember that the Omega is more suitable when construct got more than one factor. Indeed, this reliability index take in consideration the scale construct (how many factors, items in those factors …). But when the scale construct is made up with one factor, omega and alpha will have close values.


References

  1. Baglin, J. (2014). Improving Your Exploratory Factor Analysis for Ordinal Data : A Demonstration Using FACTOR. Practical Assessment, Research, and Evaluation, 19(5), 1‑16. https://doi.org/10.7275/dsep-4220
  2. Bagozzi, R. R., & Yi, Y. (1988). On the evaluation of structural equation models. Journal of the Academy of Marketing Science, 16(1), 74‑94. https://doi.org/0092-0703/88 / 1601-0074
  3. ten Berge, J. M. F., & Kiers, H. A. L. (1991). A numerical approach to the approximate and the exact minimum rank of a covariance matrix. Psychometrika, 56(2), 309-315. http://doi.org/10.1007/BF02294464\
  4. Buja, A., & Eyuboglu, N. (1992). Remarks on Parallel Analysis. Multivariate Behavioral Research, 27(4), 509-540. http://doi.org/10.1207/s15327906mbr2704_2\
  5. Devlin, S. J., Gnanadesikan, R., & Kettenring, J. R. (1981). Robust estimation of dispersion matrices and principal components. Journal of the American Statistical Association, 76, 354-362. http://doi.org/10.1080/01621459.1981.10477654\
  6. Ford, J. K., MacCALLUM, R. C., & Tait, M. (1986). The application of exploratory factor analysis in applied psychology : A critical review and analysis. Personnel Psychology, 39(2), 291‑314. https://doi.org/10.1111/j.1744-6570.1986.tb00583.x
  7. Garrido, L. E., Abad, F. J., & Ponsoda, V. (2013). A new look at Horn’s parallel analysis with ordinal variables. Psychological Methods, 18(4), 454‑474. PubMed. https://doi.org/10.1037/a0030005
  8. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1‑55. https://doi.org/10.1080/10705519909540118\
  9. Lee, S.-Y., Poon, W.-Y., & Bentler, P. M. (1995). A two-stage estimation of structural equation models with continuous and polytomous variables. British Journal of Mathematical and Statistical Psychology, 48(2), 339‑358. https://doi.org/10.1111/j.2044-8317.1995.tb01067.x
  10. Peterson, R. A. (2000). A Meta-Analysis of Variance Accounted for and Factor Loadings in Exploratory Factor Analysis. Marketing Letters, 11(3), 261‑275.
  11. Sharma, S., Mukherjee, S., Kumar, A., & Dillon, W. R. (2005). A simulation study to investigate the use of cutoff values for assessing model fit in covariance structure models. Journal of Business Research, 58(7), 935‑943. https://doi.org/10.1016/j.jbusres.2003.10.007\
  12. Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the Fit of Structural Equation Models : Tests of Significance and Descriptive Goodness-of-Fit Measures. Methods of Psychological Research, 8(2), 23‑74
  13. Timmerman, M. E., & Lorenzo-Seva, U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16(2), 209-220. http://doi.org/10.1037/a0023353\
  14. Tinsley, H. E. A., & Tinsley, D. J. (1987). Uses of factor analysis in counseling psychology research. Journal of Counseling Psychology, 34(4), 414‑424. https://doi.org/0022-0167.34.4.414\
  15. Trizano-Hermosilla, I., & Alvarado, J. M. (2016). Best Alternatives to Cronbach’s Alpha Reliability in Realistic Conditions : Congeneric and Asymmetrical Measurements. Frontiers in Psychology, 7(769). https://doi.org/10.3389/fpsyg.2016.00769\
  16. Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach’s α, Revelle’s β, and Mcdonald’s ωH : Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123‑133. https://doi.org/10.1007/s11336-003-0974-7\

CFA

Row

Commentary

The same criterion as the EFA are used. The TLI index must be higher than .95, the RMSEA must be lower than .10, the SRMR must be lower than .08, the chi2 should be non significant, the GFI index should be higher than .95, the AGFI index must be higher than .90 and the CFI index must be higher than .90



Row

Cut-off Criterions CFS

TLI RMSEA SRMR Chi2 p.value GFI AGFI CFI
0.8671831 0.1159691 0.093122 307.4565 0 0.9680991 0.9261553 0.8913316
0.1035497
0.1287402

Row

Cut-off Criterions AQ

TLI RMSEA SRMR Chi2 p.value GFI AGFI CFI
0.5884802 0.1332097 0.1177515 251.7531 0 0.9414338 0.874501 0.6799291
0.1180058
0.1489220

Row

Cut-off Criterions RTC

TLI RMSEA SRMR Chi2 p.value GFI AGFI CFI
0.9337185 0.1505368 0.0541501 17.8176 0.0001352 0.9948545 0.9331087 0.9779062
0.0915726
0.2181803

Correlations

Row

Commentary

The pearson correlation on item sum is used to assess the scale validity. We hypothesis that our scale construct would be positivility correlated with the AQ subscale (attention switching), negatively correlated with the WCST and cognitive rigidity subscale, and that there would be no correlation between our scale and the CFS



Row

Plot WCST

Plot CFS

Row

Plot AQ

Plot RTC